System for creating a composite image and methods for use therewith

ABSTRACT

A system includes a video device for capturing, at a viewing time, a first video image corresponding to a foundation scene at a setting, the foundation scene viewed at the viewing time from a vantage position. A memory stores a library of image data including media generated at a time prior to the viewing time. A vantage position monitor tracks the vantage position and generating vantage position of a human viewer. A digital video data controller selects from the image data in the library, at the viewing time and based on the vantage position data, a plurality of second images corresponding to a modifying scene at the setting, the modifying scene further corresponding to the vantage position. A combiner combines the first video image and the plurality of second images to create a composite image for display.

CROSS REFERENCE TO RELATED PATENTS

This application is a continuation of U.S. patent application Ser. No.16/297,333, filed Mar. 8, 2019, which is a continuation of U.S. patentapplication Ser. No. 15/079,166, filed Mar. 24, 2016, now U.S. Pat. No.10,230,905, which is a continuation of U.S. patent application Ser. No.14/323,179, filed Jul. 3, 2014, now U.S. Pat. No. 9,311,753, which is acontinuation of U.S. patent application Ser. No. 12/626,603, filed Nov.25, 2009, now U.S. Pat. No. 8,817,092, which claims benefit ofprovisional U.S. Patent Application No. 61/117,949, filed on Nov. 25,2008, which are hereby incorporated by reference herein in theirentirety.

TECHNICAL FIELD OF THE INVENTION

This invention relates to a method and apparatus for generating andviewing combined images of viewed locations. The invention hasparticular application to enhancing the viewing experience of touristsat historically significant locations.

DESCRIPTION OF RELATED ART

At a historically significant location, it is usual to find some sort ofsurviving artifact—for example, a ruined temple, a harbour wall, or acastle keep—which remains to show that the location was in facthistorically significant. For example, the temple was the site ofworship and sacrifice, or the harbour saw the provisioning of woodenwarships, or the keep was part of a castle that underwent a long siegebefore falling to attackers. For a tourist who will spend an hour or twoat the site, some sense of the historical significance of the site canbe gained by consulting a guidebook, or listening to a tour guide who ispresenting either in real time or by means of an audio transcript. Whilea guidebook or audio aid enhances the tourist experience, improvementsin the delivery of relevant information are possible.

In this respect, United States patent application 2006/0271292 describesa wearable geographical information display assembly including aheads-up display to which is mounted an attitude sensor. Both theheads-up display and the attitude sensor are coupled with a cellularphone of a type that receives geographical positioning data from orbitalsatellites. Positional data and data indicating line of sight from theattitude sensor are transmitted from the cell phone to atelecommunications company. The company retrieves display datacorresponding to the received positional and line of sight data from adatabase. The display data is transmitted back to the cell phone anddisplayed by means of the heads-up display.

While this arrangement may be effective in presenting to the tourist acomprehensive description of what is being viewed and its significance,further improvements in the preparation and delivery of relevantinformation are possible which can improve a tourist's appreciation ofhistorical sites.

BRIEF DESCRIPTION OF THE DRAWINGS

It will be appreciated that for simplicity and clarity of illustration,elements illustrated in the following FIGs. have not necessarily beendrawn to scale. For example, the dimensions of some of the elements areexaggerated relative to other elements for clarity. Other advantages,features and characteristics of the present disclosure, as well asmethods, operation and functions of related elements of structure, andthe combinations of parts and economies of manufacture, will becomeapparent upon consideration of the following description and claims withreference to the accompanying drawings, all of which form a part of thespecification, wherein like reference numerals designate correspondingparts in the various figures, and wherein:

FIG. 1 shows a perspective view of a tourist location and a touristviewing the location.

FIG. 2 shows a schematic view of a part of each of a foundation,modifying and combined image produced in the course of operatingequipment according to one aspect of the invention.

FIG. 3 shows a schematic diagram of a system according to one embodimentof the invention.

FIG. 4 shows stages in a background suppression routine forming part ofa method according to one embodiment of the invention.

FIG. 5 shows one embodiment of a heads up viewing device embodying theinvention.

FIG. 6 shows another embodiment of a heads up viewing device embodyingthe invention.

FIG. 7 shows a part of a field of view showing registration sight marksdeveloped using a method according to one embodiment of the invention.

FIG. 8 is a top view of a person wearing a heads up viewing deviceaccording to an embodiment of the invention.

FIG. 9 shows a person wearing a heads up viewing device according to anembodiment of the invention and demonstrates one aspect of viewervantage position.

FIG. 10 shows a part of a field of view showing registration sight marksdeveloped using a method according to one embodiment of the invention.

FIG. 11 shows a person wearing a heads up viewing device according to anembodiment of the invention, and demonstrates another aspect of viewervantage position.

FIG. 12 shows a person wearing a heads up viewing device according to anembodiment of the invention, and demonstrates a further aspect of viewervantage position.

FIG. 13 shows a part of a field of view showing registration sight marksdeveloped using a method according to one embodiment of the invention.

FIG. 14 shows a schematic diagram of a viewing sub-system according toone embodiment of the invention.

FIG. 15 shows a schematic diagram of a recording sub-system according toone embodiment of the invention.

FIG. 16 shows a schematic view of a part of each of a foundation,modifying and combined image produced in the course of operating asystem according to one aspect of the invention.

FIG. 17 shows a heads up viewing device according to an embodiment ofthe invention.

FIG. 18 shows a schematic view of parts of modifying images produced inthe course of operating the system of FIG. 17 .

FIG. 19 shows a heads up viewing device according to another embodimentof the invention.

FIG. 20 shows a schematic view of parts of modifying images produced inthe course of operating the system of FIG. 19 .

FIG. 21 shows a view from above of recording camera and range finderfunction according to an embodiment of the invention.

FIG. 22 shows a view from above of a viewer wearing a heads up viewingdevice according to an embodiment of the invention.

FIG. 23 is a view from above showing superinposition of objects of amodifying and a foundation image developed upon combination of theimages according to one embodiment of the invention.

FIGS. 24 through 26 show image fragments developed during an unwantedobject suppression routine forming a part of a method forming oneembodiment of the invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS OF THE INVENTION

In this specification, “vantage position” means the combination of anyone or more of a set of parameters comprising position (x, y, z),direction of viewing (i.e. the combination of azimuth and elevation),tilt (corresponding to roll in the dynamic sense), and depth of focus,that determine the position and attitude of a person or viewing devicewhen viewing a scene. In this specification, a “foundation image” is animage of a foundation scene which is viewed at a location or setting,either directly or by means of a camera or other viewing device, at aviewing time or phase; a “modifying image” is an image of objects thatdo not exist at the location or setting at the viewing time, but whichhave existed at a prior recording time and has been previously recordedas video data and, optionally, videographics digital data in the courseof a production phase.

Referring to FIG. 1 , there is shown a historically significant location2 which in this case is an ancient temple which is at least partiallyintact or restored. A tourist 4, at a different scale from the temple inthis FIG., views the location from a vantage position, and from thatvantage position, sees a foundation image 6 of the location, part of thefoundation image being shown in the upper right circular frame in FIG. 2and being a front view of several temple columns. The tourist wears aheads-up viewing device 8 by means of which she is viewing a combinationof the foundation image 6 and a modifying image 10, part of themodifying image being shown in the upper left circular frame. Part ofthe combined image 12 is shown in the lower circular frame in FIG. 2 .The modifying image 10 is generated as if it were being viewed from thesame vantage position from which the tourist 4 views the foundationimage 6. As shown in FIG. 2 , elements of the modifying image 10 arecombined with elements of the foundation image 6 at the heads-up viewingdevice 8 so that the tourist sees the elements of the modifying imagelocated immediately in front of the temple columns.

Referring to the schematic diagram of FIG. 3 , a system according to oneembodiment of the invention has a number of functional blocks formingpart of a production phase sub-system 14 and a number of functionalblocks forming part of a viewing phase sub-system 16. The productionphase sub-system 14 includes either or both of a video camera 18 torecord a preliminary modifying image of a modifying scene, or avideographics software development function 20 to artificially constructa modifying image 10 corresponding to a notional modifying scene. In theparticular production phase sub-system 14 illustrated in FIG. 3 , both avideo camera 18 and videographics software development function 20 arepresent, the combination being operable to generate modifying images 10which are combinations of video and videographics content. The videocamera 18 or videographics development software 20 produces recordedcontent that is digitized and stored in memory 22 as video stream data.When using the video camera 18, a modifying image 10 of a modifyingscene is normally recorded at the site of the foundation scene,preferably using certain artifacts of the foundation scene to placeobjects of the modifying scene for recording by the recording videocamera 18. However, the modifying image 10 can be recorded by the videocamera 18 at a location remote from the foundation scene provided thatobjects in the modifying scene are so placed that objects in themodifying image 10 and in the foundation image 6 will be properlylocated relative to each other when viewed as the combined image 12.

Associated with the video camera 18 is a camera vantage positionmonitoring function 24 for monitoring the instant value of certainoperating parameters of the video camera 18, these being any or all ofthe video camera x,y,z position, viewing direction axis, tilt, and depthof focus. An output from the monitoring function 24 is digitized andadded as metadata to the video stream data stored in memory 22 so thatdata for each stored video stream has associated with it synchronizedcamera vantage position metadata. For a modifying image 10 generatedusing the videographics software 20, corresponding videographics vantageposition metadata is generated by a videographics vantage positionfunction 25 and stored in the memory 22 with the digital videographicsdata, the vantage position metadata being generated directly fromroutines within the videographics software 20.

The viewing phase sub-system 16 includes the heads-up viewing device 8of FIG. 1 which is operable to display a combination of elements from amodifying image 10 and elements from a foundation image 6 which arecombined at an image combiner 26. The foundation image 6 is generatedfrom a foundation scene 28 solely or predominantly during the viewingphase, either at a video camera 30, from which the foundation image isprovided to and displayed at the heads-up viewing device 8 (“videofoundation”), or as a real space image observed via the heads-up viewingdevice 8 (“optical foundation”). Use of a video foundation image, whilemore complex in operational terms, has some advantage over use of anoptical foundation image in allowing a greater degree of imagemanipulation as will be described presently. In the viewing phase, oneor more parameters of the viewer's vantage position are identified at aviewer vantage position monitor 32. On the basis of the vantage positionparameter data, corresponding modifying image data is retrieved frommemory 34 and is used to generate a modifying image at a modifying imagegenerator 36. The modifying image 10 is combined with the foundationimage 6 at the image combiner 26 and the combined image is viewed by theviewer at the heads-up viewing device 8.

Modifying image data stored in memory 22 and/or 34 may be video data,videographics data or a combination of video data and videographicsdata. The particular modifying image data retrieved from memory dependson the nature of viewer vantage position and other data associated withthe viewer and the viewing activity, so that the modifying image 10developed corresponds in time and apparent vantage position to theinstant vantage position of the viewer. The modifying image 10 may bestill, such as an image of buildings, or moving, such as an image ofpeople working.

The modifying image 10 and the foundation image 6 can be considered aseach comprising certain objects and background. Background to be viewedas part of the combined image 12 is normally delivered as part of thefoundation image 6 while background present in the initially generatedmodifying image 10 is normally suppressed. As shown in the top leftrectangle of FIG. 4 , an initial modifying image 10 is assumed to havebeen generated by recording video camera 18, the initial modifying imagecontaining a desired foreground object 38 in the form of a standingperson and an undesired background 40 which is here shown as trees andsun. In preparation for suppressing the unwanted background from themodifying image 10, the object 38 intended to remain in the modifyingimage is marked for example with small areas of colour of a definedwavelength distributed over the entire object surface. An objectsdetection sensor function 42, shown in the FIG. 3 schematic diagram, andmounted with the video camera 18, receives a component of the modifyingscene being recorded by the video camera and senses the existence andlocation of the marked areas signifying the desired foreground object38. Under a control signal generated from the objects detection sensorfunction 42, the recording video camera 18 generates an image of themarked object 38. At other parts of the scene, where no marker output issensed, a monochrome background development function 43 develops amonochromatic portion 44 as shown in the top right rectangle of FIG. 4 .Subsequently, during the viewing phase, the monochromatic portion 44within the modifying image data is detected and suppressed at abackground suppression function 45 shown in FIG. 3 so that in thecombined image displayed to the viewer (bottom right rectangle of FIG. 4), the viewer sees background content from the foundation image (bottomleft rectangle). In the case of a modifying image developed usingvideographics software, an image is developed that consists solely ofdesired foreground objects and no suppression step is needed. All areasnot occupied by foreground objects are shown as monochromatic areas tobe replaced by foreground from the foundation image.

In one embodiment of the invention, all processing capability, memoryand stored data for generating available modifying images are carried byor mounted on the viewer. In another embodiment of the invention,minimal processing capability and memory are present in equipmentcarried by the viewer and a transmission path exists between the viewerand a remote server location or locations, where a significant amount ofthe processing capability and memory for the viewing phase are situated.For example, each of the viewer and the server locations can include awireless transceiver forming part of a wireless network between theviewer and the server location. The wireless network is used to relaydigital video data and vantage position and command data between theviewer and the server. A number of alternatives are possible inallocating memory, processing capability and stored data as betweenequipment at the viewer, equipment at one or more locations at the siteof the foundation image, and equipment at a remote site. In addition,networks for conveying video data and vantage position data can includewired and wireless networks. Video data may be transmitted from a serversite in real time as and when needed by a viewer or may be downloadedand locally stored as the viewer enters the foundation site or aparticular part of the foundation site.

Referring to FIG. 5 , one form of a heads-up viewing device 8 for use inthe viewing phase sub-system is shown. This has the form of modifiedeyeglasses and is functionally similar, in some respects, to theeyeglasses described by Buck in Patent Application Publication2006/0271292, which is hereby incorporated by reference in its entirety.The modified eyeglasses have a frame 48 to which a heads-up imagingsystem 50 is mounted. One example of a heads-up imaging system is a640×480 colour VGA capable heads-up display unit, the Second SightM1100, from Interactive Imaging Systems, Inc. of Rochester, N.Y., thissystem including a display 52 which mounts to one of the lenses 54 ofthe modified eyeglasses. Another heads-up display is the iWear VR920from Vuzix Corporation of Rochester, N.Y. While adaptations of suchheads-up viewing devices may be suitable for certain embodiments of theinvention, more complex heads-up viewing devices may be required forother of the embodiments described. Addressable display technology canbe of any suitable technological type including LCDs, MEMs, etc. As analternative to the eyeglasses form of heads-up display in which themodifying image is generated at an addressable display positioned at aconventional lens position of eyeglasses, the modifying image may beprojected onto a partially reflecting screen to be viewed by the viewer,and the viewer looks through the screen to see the foundation scene. Inyet another alternative, the addressable display technology can beimplemented in a contact lens as described by Parviz et al., Universityof Washington, at the November, 2009 Biomedical Circuits and Systemsconference in Beijing, China

Also mounted to the eyeglasses frame 48 is a sensor system 56 formingpart of the vantage position monitor 32 for monitoring a viewer'svantage position, meaning any one or more of position (x,y,z), viewingdirection, head tilt, and depth of focus. To measure position, thesensor system has mounted within it a receiver which receives data froma high quality differential GPS (DGPS) or from local terrestrialtriangulation beacons.

The sensor system 56 has an output to a central processing unit 58 forcomputing the position of the viewing device from the received GPS ortriangulation data. To measure viewing direction, the sensor system 56has mounted within it a compass which detects magnetic north and fromthis reference, azimuth or bearing angle is computed. A first leveldetector mounted in a plane transverse to the device viewing axisdetects head tilt by sensing inclination from level of the viewingdevice. A second level detector mounted in a vertical plane containingthe viewing axis senses elevation. Outputs corresponding to each of themeasured parameters are developed from the respective sensors. Theparticular design and inter-relationship of sensors forming part of thesensor system is not critical and the sensor system 56 is shown in FIG.5 as a composite block mounted to one leg of the eyeglasses frame 48.

As shown in an alternative embodiment illustrated in FIG. 6 , theviewing device may also include an eyeball analyzer forming part of aneye gaze direction monitor 60. One form of eye gaze direction monitor isdescribed in U.S. Pat. No. 4,595,990 (Garwin et al.) which refers to anearlier article by Merchant et al., “Remote Measurement of Eye DirectionAllowing Subject Motion Over One Cubic Foot of Space”, published in IEEETransactions in Biomedical Engineering, BME 21, No. 4, July 1974, pp.309-317, the patent and the article each being incorporated herein byreference in their entirety. An output from the eye gaze directionmonitor 60 is used to generate a first sight point 62 in the viewer'sfield of view as represented in FIG. 7 , this being indicative of theviewer's eyeball gaze direction as shown by arrow A in FIG. 8 showing aview from above of the viewer's head. An output from the vantageposition monitor 56 is used to generate a second sight point 64 in theviewer's field of view, this being a point on the axis of the viewingdevice 8 and therefore indicative of the viewing direction as shown bythe arrow B of FIG. 7 . These two sight points are presented in theviewer's field of view and when lined up by the viewer adjusting hisdirection of gaze and/or the axis of the viewing device 8, signify thatthe viewer's eyeball gaze direction is aligned with the axis of theviewing device. During a viewing phase, the viewer seeks to keep the twosight points 62, 64 aligned to ensure that the modifying and foundationimages will be properly matched in terms of respective vantagepositions. The sight points are represented to the viewer not in such away as to be unduly intrusive or to detract from the quality of thecombined image when viewed.

The viewer vantage position monitor may include a depth-of-focus monitorincluding an eyeball analyzer which may be the same eyeball analyzer asis used for monitoring eyeball gaze direction and sharing certain sensecircuit elements. In the depth-of-focus case, as known in the art, theeyeball analyzer has a sensor to detect changes in the spherical radiusof the eyeball lens. An output from the depth-of-focus sensor forms afurther component of the vantage position sensor system.

Other more or less sophisticated sensor systems for measuring viewingdirection are possible and there is no intention to limit theapplication and construction of the invention to any particular sensordesign. For example, in another form of sensor system, also illustratedas part of the alternative viewing device of FIG. 6 , three or morespaced detectors 68 are positioned at spaced locations about the viewingdevice 8. The spaced detectors 68 each accurately monitor x,y,zpositional data received from local terrestrial triangulation beacons.Viewing direction is then computed by appropriately comparing x,y,zvalues of the spaced detectors.

In certain embodiments of the invention, there may be circumstanceswhere not all of the components of vantage position may need to bemonitored: for example, the viewer is required always to maintain histwo eyes level with each other (i.e. no head tilt); or alternatively,the system is to operate only in a single horizontal and/or verticalplane and therefore there is no requirement to measure the vantageposition parameters corresponding to elevation and/or azimuthrespectively. In such systems, certain components of the sensor systemcan be obviated or modified so as to operate solely to present sightingindicators in the viewer's field of view as illustrated in FIGS. 9-13 .For example, and referring to FIGS. 9 and 10 , if the viewer isconstrained always to look at the foundation scene with head heldvertically, there is no need for a head tilt parameter to be generatedand used in generating the modifying image 10. In such an embodiment,the head tilt sensor output is used to generate and to present in theviewer's field of view a pair of sight lines, one line 70 aligned withthe viewer's head tilt and the other line 72 aligned with a verticalplane containing the viewing device viewing axis as shown in circularframe of FIG. 10 representing the viewer's field of view. During theviewing phase, if the sight lines are misaligned as shown, thisindicates that the viewer has her head on a tilt. In order to view thecombined image 12 with the foundation image 6 and modifying image 10 incorrect registration, the viewer adjusts her head tilt position untilthe two sight lines 70, 72 are aligned.

In another embodiment of the invention, the sensor system includes onlypositional (x,y,z) sensors. A modifying image 10 initially delivered tothe viewer is selected on the presumption that the viewer's viewingdirection is level (zero elevation) and has a desired bearing, X. Asshown in the side view of FIG. 11 , the viewer may initially be lookingupwardly and, as shown in the top view of FIG. 12 , may be looking at anangle to desired bearing 74. As represented in FIG. 13 , “cross hairs”76, 78 are generated using the vantage position sensor sub-system 56,one cross-hair 76 representing the position of zero elevation and thedesired bearing and the other cross hair representing the actualelevation and angle away from the desired bearing of the viewer'scurrent viewing direction. Any misregistration of the modifying image 10with the foundation image 6 is then corrected by the viewer bringing thetwo “cross hairs” 76, 78 into registration. Clearly, the head tilt sightregistration of FIGS. 9 and 10 can be combined with the sightregistration scheme of FIGS. 11, 12 and 13 . In a variation of thismethod, the sight lines and cross-hairs are not used. Instead, theviewer simply changes his head tilt and/or viewing direction until theviewed foundation image 6 appears to his perception to be inregistration with objects in the modifying image 10.

As illustrated in the embodiments of FIGS. 5 and 6 , the sensor systemis mounted to the eyeglasses type viewing device 8 which is convenientbecause any sensed change in vantage position of the viewing deviceequates inevitably to a corresponding change in vantage position assensed by the vantage position sensor system 56. However, the sensorsystem can be mounted at another part of the viewer's head provided thatthere is a substantially unchanging registration as between the viewer'shead and the sensor system 56 and the viewer's head and the viewingdevice 8.

In use, a viewer dons the heads-up modified eyeglasses 8. Aninitialisation phase at power-up may include a calibration sequence inwhich certain properties or characteristics of the viewer and/or viewingenvironment are monitored or measured, as applicable. Such a calibrationsequence may, for example, include performance of a test sequence torelate viewer eyeball lens spherical radius to selected depth of focus.The calibration sequence may also involve monitoring the height of theviewing device with the viewer standing on the ground. The calibrationmay also involve monitoring ambient and non-ambient light, etc. Suchcalibration may be used to determine certain video data streams thatwill not be needed during the viewing phase and may be used to determinecertain video data streams that are highly likely to be needed duringthe viewing phase.

The wireless transceiver establishes communication with the wirelessnetwork and transmits the calibration data and initial vantage positiondata to a server location. On the basis of the received vantage positiondata, a digital video data controller retrieves corresponding digitalvideo data from the memory at the server location and sends a video datastream to the viewer. The retrieved digital video data are datapreviously generated and recorded in the production phase, and may bevideo data, videographics data, or a combination of the two. Theretrieved data is transmitted from the server location over the wirelessnetwork and is received by the viewer transceiver where it is used togenerate the modifying image. The modifying image 10, elements of whichare presented to the viewer, corresponds in vantage positional terms tothe foundation image 6 being viewed by the viewer. Elements of themodifying image and the foundation image are combined at the heads-upviewing device 8.

In one embodiment of the invention, the viewer may have reviewed a menuof possible performance acts, respective acts having content consideredby historians as being appropriate to particular eras. The viewer maymake a selection from the menu which causes a command signal to begenerated to call up a modifying image depicting the selected act. Theselected act may be still: for example, it shows modifying imagearchitectural features that may have existed in an earlier centurycombined with present architectural features derived from the foundationimage. Or the selected act may be a moving image.

The command signal is transmitted to the video memory to initiateretrieval of a particular data stream for the selected act. The viewertypically elects to view the act from the beginning although, if theviewer has seen part of the act, then she may instruct video datastreaming to start at some intermediate part of the act. As part of theinitialization phase, the parameters of the present vantageposition—some or all of x,y,z position, viewing direction, head tilt,and depth of focus of the viewing device—are monitored. Datacorresponding to monitored parameters are then generated and transmittedto the memory controller. The memory controller determines what videodata stream is developed so that the image viewed by the viewer is theappropriate image for the viewer's then-current vantage position.

The selected video data is the data corresponding to the present vantageposition. No data associated with any other vantage position isaccessed, except possibly as a result of running certain prediction,interpolation, etc., algorithms as will be described presently.

In one embodiment, the viewer is constrained to view a single combinedvideo image corresponding to one vantage position (a single position,viewing direction, head tilt attitude and depth-of-focus) and adopts andmaintains that particular vantage position during the course of theviewing phase. In another embodiment of the invention, a modifying imageis presented for each of a series of vantage positions, the modifyingimage received at one vantage position differing from the modifyingimage received at a different vantage position. As the viewer moves, newviewer vantage position data is sent from the viewer to the server. Atthe server, the new vantage position data are analyzed. In response, avideo stream corresponding to the new vantage position is retrieved andsent from the server over the wireless network to the viewer, where itis used to generate a different modifying image. Provided that changesto the modifying image are rapid and accurately accord with changes inthe viewer's vantage position, this can give the viewer the impressionthat she is moving around or through the combined tableau.

In addition to basic system functions depicted in the schematic blockdiagram of FIG. 3 , the system may have several optional functionsdepicted by further blocks shown in FIGS. 14 and 15 , such optionalfunctions being used to achieve more sophisticated performance andfeatures at the cost normally of greater complexity of the productionand viewing sub-systems, and more processing in the production andviewing phases. Some of the optional functions may be solely embodied inone or other of the production and viewing phase sub-systems, whileothers may have components within both sub-systems.

Referring to FIG. 14 , the viewing phase sub-system includes a modifyingact selector/timer 80. This function enables the tourist to choose aparticular act to view as a modifying image. For example, at ahistorical site, the viewer may wish to view a video act typical of oneearlier century and then view an act typical of a later century. Or theviewer may want to see an enactment of a particular event, such as abattle. The selector/timer 80 may include a control available to theviewer to start, stop, fast forward and reverse the performance of anysuch act.

The viewing phase sub-system optionally includes a manual vantageposition command function 82. Such a function provides an alternative oradditional method for generating vantage position control data incomparison with generating the vantage position control data using thesensor system 56 described with reference to FIGS. 5 and 6 . By means ofthe manual vantage position command function, the viewer operates amanually operated controller to transmit any or all of shift up/down,shift left/right, rotate clockwise/anticlockwise, focus in/out commandsignals to a receiver where the received signal is processed and used toselect the modifying image presented at the viewing device 8.

The viewing phase sub-system optionally includes a sight marks generatorfunction 84. Such a function was described with reference to theembodiments of FIGS. 9-13 and is used to generate sight marks associatedwith the difference in eye gaze direction and viewing direction (FIGS. 9and 10 ) and/or with pre-registration positions of the modifying imageand the foundation image (FIGS. 11-13 ). By appropriate adjustment ofvantage position, the viewer can bring particular pairs of viewed sightmarks or lines into registration, at which point she knows that she hasachieved registration between the modifying image and the foundationimage.

The use of vantage position data or manual control to position themodifying image in the viewer's field of view may result in somepositional misregistration of the modifying image with the foundationimage. For example, the modifying image may be translationally orangularly displaced relative to the foundation image. Or the modifyingimage may be distorted as by stretching or contracting in one of moredirections relative to the foundation image. Such misregistration mayoccur as the registration process nears completion or after the primaryregistration process is completed. In both situations, such minormisregistration is corrected using an optional locking function 86 inthe viewing phase sub-system. One example of such a locking functiondepends on having recorded a number of registration positions in themodifying image 10 during the production phase. For example, when themodifying image 10 is being recorded by the recording video camera atthe site of the foundation scene, a receiver at the video camerareceives radio locking data transmitted from a number of radiotransmitter beacons spaced around the foundation scene as depicted byimage positions 88 shown in the recorded image at the top left circularfield of view frame illustrated in FIG. 16 . The locking data isrecorded as metadata associated with the corresponding modifying imagedata. The same radio beacons also transmit positional data during thesubsequent viewing phase when the foundation image 6 is being viewedthrough the viewing camera, and are depicted as image positions 90 inthe top right circular frame of FIG. 16 . The beacon signals arereceived at the viewing phase camera and processed at a registrationunit. If the positions of the respective beacons are mismatched asbetween their positions in the foundation image and their positions inthe modifying image, such mismatch being depicted in the lower circularfield of viewe frame of FIG. 16 , a locking registration adjustment isapplied to the modifying image. As indicated, this can change theposition of the modifying image translationally, angularly, or in otherways, as necessary.

Ideally, for a viewing experience offering a high measure ofverisimilitude or virtual reality, an available modifying image shouldbe accessible for every vantage position that might be adopted by aviewer during a viewing phase within a viewing distance of thefoundation scene. Clearly, such an image library for a particularlocation connotes an enormous level of available stored digital videodata and associated processing both in terms of the viewing session andthe production session. For a practical system, the stored datarequirement is reduced by having a limited number of vantage positionsfor which corresponding modifying image data is stored.

At its simplest, the invention is implemented with a single vantageposition with one viewing position, one viewing direction and onedepth-of-focus. In contrast, the viewing phase sub-system of FIG. 14includes an optional zone limiting function 92, the operation of whichdepends on any or all of the parameters monitored by the vantageposition monitor: i.e. x,y,z position, viewing direction, depth-of-focusand head tilt. The zone limiting function 92 determines whethermodifying image data exists for the particular monitored viewer vantageposition. If it does not, an output from the zone limiting functionprevents data selection from the modifying images data memory and theviewer receives a visual or audible notification that no modifying imagecan be received by the viewer. A number of vantage positions for whichvideo data is stored and available can be clustered in zones at thefoundation site so that combined images are available to the viewerinside, but not outside, a cluster zone. During a viewing phase, theviewer may move from a combined image display zone, through a no-displayzone, to a further combined image display zone. Alternatively, for acluster of closely adjacent active vantage positions, there can be asmooth transition as the viewer walks through the cluster zone, and acombined image associated with one vantage position transitionsrelatively seamlessly to a combined image for an adjacent vantageposition. The vantage positions may also be clustered as viewingdirections within a defined angular viewing cone.

As indicated previously, a large modifying image library requires largememory and significant processing capability. Particularly for a low orzero action modifying image, memory and processing requirements arereduced by having certain modifying images presented only as a series ofstill images with a period of time between presenting each still image.As shown in FIG. 15 , memory requirements can be further reduced at theexpense of additional processing by utilizing a video compressionfunction 94 in the production phase and a corresponding videodecompression function 96 in the viewing phase. Various compressiontechniques are known and used in video coding standards such as MPEG-2,AV-1, etc, and such techniques, or adaptations of them, can be utilizedin storing digital video data for embodiments of the present invention.In most conventional video applications, video compression algorithmsare designed to have greatest efficacy when compressing single streamvideo. Storage of video data for modifying images according to thepresent invention lends itself to additional compression modes, sinceredundancy exists as between images at different but closely adjacentpositions, at different but closely adjacent viewing directions, atdifferent but closely adjacent depths of focus, and at different butclosely adjacent time intervals. Compression schemes tailored to thesesparallel redundancies within neighbouring video streams are possible.

In another embodiment of the invention, an audio playback function 98forms a part of the viewing phase sub-system as shown in FIG. 14 and anaudio recording function 100 forms part of the production phasesub-system as shown in FIG. 15 . In one embodiment of such audiorecording and playback functions, digital audio data and sound trackmetadata are integrated with digital video data comprising the modifyingimage data streams and the audio playback function 98 includescorresponding digital audio data decoding and audio generationequipment. The audio playback function 98 can be adapted to receive anddecode a sound track that is made vantage position dependent bygenerating, in the course of the production phase, several sound tracksfor each performed act corresponding to the sounds that might be heardat spaced locations in the act being depicted in the modifying image. Asthe viewer moves between different positions or focuses on differenttableaux in the foundation scene, one sound track may be displaced byanother. In a variation, appropriate audio control data is included inthe metadata so that during a viewing phase, parallel sound tracks areadjusted in relative volume to present the verismilitude of asoundscape. As the viewer passes between different tableaux within thehistorical scene, the soundscape changes, with one or more sound tracksfading as other sound tracks are amplified. In certain embodiments ofthe invention, audio data is integrated with a digital video stream and,it will be understood that the term video stream, as used in thisspecification, encompasses a video data signal that has such an integralaudio component. In other embodiments of the invention, some or all ofthe available audio streams may exist independently of the video streamand retrieval of audio data may be under a viewer control function thatis separate from the viewer's control of video data retrieval. Forexample, such an audio control operates to retrieve and deliver to theviewer an audio stream that is a function of the viewer's viewingdirection so that the audio received by the viewer does not depend onwhere he is positioned, but at what he is looking. Such an audio streamcan also include a selectable commentary so that the viewer can choosewhether, for example, to hear a commentary related to the combined imagebeing viewed or a combination of such a commentary and a soundscapecorresponding to the image. In yet another arrangement, as opposed to orin addition to audio delivered to the viewer, text description relatedto what is being viewed is delivered to the heads-up viewing device,either integrated with the digital video stream or as a parallel stream.

In another embodiment of the invention, the viewing phase sub-system ofFIG. 14 has an optional stereoscopic viewing function 102. As part ofsuch a function, a modified eyeglasses heads-up viewing device 8 isformed with separately addressable displays 104 fabricated at each lensposition as shown in FIG. 17 . The stereoscopic function includes anaddressing circuit operable to deliver modifying images individually toeach of the viewer's eyes. As shown in FIG. 18 , which depicts imagesbeing presented to the viewer's left and right eye displays, themodifying image delivered to the left eye display is marginallydifferent from the modifying image delivered to the right eye display,the two video data streams corresponding to the vantage positionsrespectively of the viewer's left eye and right eye. By presentation ofleft and right modifying images, the viewer perceives a stereoscopicimage giving the impression of depth. In this respect, attention isdrawn to U.S. Pat. No. 4,984,179 (“the '179 patent”) which isincorporated herein by reference in its entirety. Although primarilydirected at a videographics technique, the '179 patent discloses aheads-up display which includes display units which present separateimages to the respective eyes of a viewer so that a combined image isperceived stereoscopically. As described in the '179 patent, andapplicable to the stereoscopic embodiment of this invention, operatoreye movement is also monitored and used to modify images presented tothe viewer to enhance the sense of realism. As shown in the FIG. 17embodiment, a viewing video camera 30 is mounted to the eyeglasses and,as in previously described embodiments, the foundation image as recordedby the viewing video camera and is combined with the particularmodifying image for presentation at the lens displays.

In a variation shown in FIGS. 19 and 20 , a modified eyeglasses heads-upviewing device 8 is used with a pair of viewing video cameras 30 mountedclose to respective eyeglass lenses. The two viewing phase video camerasgenerate a left eye foundation image and a right eye foundation image,with parts of the two foundation images being delivered to respectiveviewing elements of the heads-up viewing device interlaced with parts ofthe modifying images delivered to the viewing elements. In this way,both the foundation image and the modifying image are viewed asstereoscopic versions to give an accentuated impression of depth.

In this particular example, the foundation images and the modifyingimages are delivered at separately addressable displays 104 occupyingpositions that would normally be occupied by a lens in a conventionalpair of eyeglasses. It will be understood that it is not intended to belimited to such a viewing device. In another embodiment of theinvention, the stereoscopic versions of the modifying image areprojected onto a reflecting screen to be viewed by the viewer, theviewer looking through the screen to see the foundation scene.

As further shown in FIG. 14 , the viewing phase sub-system optionallyincludes a stable position detector 106 which has an output used to cutoff the display of a modifying image and to instead show a clear displaywhen the viewer is moving in a translational sense, such a cut off beingof value for navigating terrain, including the presence of other people,without collisions. The viewing phase sub-system also optionallyincludes a rate of change threshold detector 108. During the course ofany vantage position change by the viewer, the viewer vantage positionsensor system 56 and digital video retrieval function are attempting tocompute instant vantage position and to lock on to the video orvideographics data feed for a corresponding modifying image 10. Themodifying image delivered to the viewer is consequently changingrapidly. In view of delay in locking onto the right corresponding sceneand the lack of settled data owing to the control system having toswitch rapidly between memory locations corresponding to different videostreams, the rate of change threshold detector has an output to suspendoperation of the viewing sub-system during periods of relatively rapidchange in viewer vantage position. The rate of change detector detectswhen there is, for any one or more governing vantage positionparameters, a rate of change more than a preset threshold. When the rateof change threshold is triggered, the modifying image viewing sub-systemshuts down for the period that the particular rate of change remainsabove the threshold. In these circumstances, the viewer is able to viewonly the foundation image through the viewing device. To inhibitoperation of the rate of change threshold detector 108, the viewer maytake care to alter her vantage position relatively slowly so that themodifying image remains visible throughout. Alternatively, the viewermay alter her vantage position relatively quickly, but then immediatelyfix her gaze at a desired target point in the foundation scene.

It will be understood that in aiming for historical verisimilitude whenviewing a foundation scene, while some objects, moving or still, may beadded to the foundation image by the modifying image, it may bedesirable to suppress other elements, moving or still, from thefoundation image.

In the embodiment illustrated in FIG. 2 , the desired objects in themodifying image overlap and obscure all objects in the foundation imagewhen the two images are combined. However, depending on vantageposition, point of time during the performance of an act, etc., someobjects to be represented in the modifying image would, in the combinedtableaux, be expected to be in front of, and at least party obscure,objects that are present in the foundation scene. Conversely, fromdifferent vantage positions, points of time, etc., some objects that arepresent in the foundation scene would, in the combined tableaux, beexpected to be in front of objects depicted in the modifying image.

In a further embodiment of the invention, as shown in FIGS. 15 and 21 ,an optional range finder function 110 is used in association with theproduction phase video camera 18 to generate range metadata for objectsthat are to appear in the modifying image; i.e., a record is made of thedistance from the camera of each of the objects which are desired to bedepicted in the modifying image and is included as metadata in thestored modifying images video streams. To effect the range finder, asthe modifying scene is scanned by the recording video camera 18, thefield of view of the camera is monitored and a computation is effectedto relate the instant scan position of the recording camera beam to aninstant viewing direction: i.e. azimuth and elevation angle off thecamera central viewing axis. Concurrently, GPS or local triangulationsignals are received from spaced transmitters 111 on each object 113which is to be viewed in the modifying image. The signal data associatedwith the objects are processed to assess each object's viewing directionfrom the camera 18. When there is a match between scan viewing directionand object beam direction, the position data of the object 113 is addedas metadata to the digital video data stream.

As shown in FIGS. 14 and 22 , a corresponding range finder function 114is used in association with the video camera 18 recording the videofoundation scene. As in the production phase, the instant scan positionof the viewing video camera 18 is related to the off axis azimuth andelevation direction. Position data is received from transmitters 115associated with objects 116 in the foundation scene and metadata isadded to the video stream which shows the range of any object from theviewing camera 18 when a match occurs between camera scan viewingdirection and the object beam direction.

As the combined image is developed, for each location in the scannedfoundation scene, range data of any object in the foundation scene atthat location is compared with any object range metadata in themodifying image data to ascertain which object is to be shown as closerto the viewer in the combined image. Then the one of the modifying andfoundation images which has the relevant object fragment apparentlycloser to the viewer is selected as the superimposed image fragment asshown in FIG. 23 .

The embodiment previously described relates to generating a modifyingimage by video recording during a production phase at the site where acombined image will later be viewed during a viewing phase. If,alternatively, the modifying image is generated by videographicssoftware, then the range, i.e. distance from the viewer, of an object tobe viewed in the modifying image, is preset by the developer. Metadatacorresponding to these ranges is added to the video data during theproduction phase and is used in the viewing phase as describedpreviously to properly arrange the objects of the modifying andfoundation images so that image data representing objects that are to beperceived as closer to the viewer appear to overlap and obscure imagedata representing objects that are to be perceived as more distant fromthe viewer.

Another embodiment of the invention optionally includes an objectsuppression function 118 as shown in FIG. 15 . Following recording andstorage of a preliminary modifying image, the image is post-processed atthe object data suppression function 118 to identify an undesired objectand to suppress data representing the undesired object and to replace itwith data representing an extrapolation of a part of the preliminarymodifying image immediately bounding the undesired object. Metadatashowing the existence of the replacement data is recorded in the digitalvideo data representing a final modifying image. As shown in FIG. 14 ,in the subsequent viewing phase, an object replacement function 120operates to detect the suppression metadata in the modifying image dataand, in response, to replace the undesired object in the foundationimage by the replacement video data tagged by the metadata. The sequenceis illustrated in FIGS. 24 to 26 in which a preliminary modifying imageas shown at FIG. 24 includes both an old thatched cottage 122 and amodern outbuilding 124. The outbuilding detracts from the foundationscene's olden day historical verisimilitude. The modifying image data ispost-processed so that as shown in FIG. 25 , image data representing theoutbuilding is replaced by replacement image data to represent anextrapolation 126 of that part of the image immediately bounding theoutbuilding. The replacement image data are tagged so that in theviewing phase and as depicted in FIG. 14 , the existence of replacementimage data is detected by a replacement image detection function 128and, under the control of an image replacement function 130, replace theoutbuilding 124 recorded in the foundation image when the modifying andfoundation images are combined as shown in FIG. 26 . Consequently, uponviewing, the appearance of the outbuilding 124 is suppressed and in itsplace is an extrapolation of the foundation scene.

The previously described embodiments of the invention are characterizedby reference positions, being vantage positions for which video streamdata has been developed and stored for generating modifying images whichare viewed as if from those vantage positions. During a viewing phase,viewing at certain unreferenced vantage positions for which no dedicatedmodifying image data has been stored may be acceptable provideddistortion arising from misregistration of the modifying image and thefoundation image is not too severe. After all, the combined image is topresent an impression as opposed to fooling the viewer into believingthat she is truly viewing a scene as it might have existed in the past.However, outside a confined zone of such unreferenced vantage positions,misregistration may be perceptually unacceptable.

In another embodiment of the invention, the viewing phase sub-systemincludes an interpolation function 132 as shown in FIG. 14 . Theinterpolation function is used to generate digital video data for anunreferenced vantage position for which no specific digital video streamdata is stored. To achieve this, the interpolation function processesstored video data corresponding to one or more reference vantagepositions close to the unreferenced vantage position using apredetermined processing algorithm. In one variation, interpolation datais generated so as to approximate vantage position data for anunreferenced vantage position intermediate a plurality of referencevantage positions. Such interpolation can be based upon any or all ofposition, viewing direction, tilt and depth of focus. Such interpolationcan be pixel based if the reference vantage positions are relativelyclose together. If the reference vantage positions are further apart,the interpolation can only practicably be based on those modifyingimages (or parts of modifying images) that are produced by videographicssoftware, whether such software is used to develop a videographicmodifying image or is used to post-process a video image to introducevideographics content and functionality into the video image.

In a variation of this embodiment, the modifying image data is generatedby processing reference vantage position data to modify the retrievedmodifying image data as a function of the viewer's detected motion awayfrom or towards a reference vantage position. One example of suchprocessing uses a distance from vantage position interpolator function134 as shown in FIG. 14 and which monitors the viewer's distance from areference vantage position (or away from a selected parameter value ofthe reference vantage position) and executes an appropriate processingalgorithm as a function of that distance to approximate the modifyingimage that should exist at that distance from the reference vantageposition. Alternatively or in addition, as shown in FIG. 14 , suchprocessing includes a rate of change interpolator function 136 whichmonitors the rate of change of such movement and executes a differentprocessing algorithm as a function of that rate of change to approximatethe modifying image that should exist at an unreferenced vantageposition on the basis of that monitored rate of change.

Referring to FIG. 15 , a video data map for a series ofcontemporaneously available modifying images must be generated in thecourse of the production phase. In one method, modifying image data arerecorded using a number of spaced production phase cameras. The camerasare, for example, fixed within an array to encompass a desired viewingzone. In one array, the cameras are distributed evenly over one form ofa notional reference surface such as a cylindrical planar surface withcylinder axis vertically disposed and with the cylinder centredgenerally on the center of the scene. In another method, one or moreproduction phase cameras are driven through a scanning sequence by acamera drive function 138. This function operates to drive each camerathrough a cycle that can involve translational and/or angular movementand/or depth of focus variation. Throughout the production phase, aspreviously described, data defining the instant recording position ofeach of the cameras is generated and is combined as metadata with thedigital video data generated by the corresponding recording phasecamera.

In the viewing phase, video stream data retrieved to generate amodifying image is limited to only that data required for the viewer'sthen-present field of view. The field of view is set partly by theviewer's depth of focus, which can be one of the monitored elements ofthe viewer's vantage position, and is otherwise defined by the viewer'snatural viewing cone which may be of the order of 80 degrees subtendedangle or less. Over 80 degrees, reduced perception of luminance,chromaticity and contrast mean that video stream data outside thisviewing cone is essentially wasted. In a further embodiment of theinvention, video data resolution is concentrated towards the centre ofthe viewing cone and reduces towards the margins of the viewing cone. Inone embodiment, outside a preset viewing cone, no modifying image datais developed. As shown in FIG. 14 , a cone limiter and concentratorfunction 140 operates to ensure that outside the preset viewing cone,the viewer perceives only the foundation image.

In a further embodiment of the invention, a light detector and adjustorfunction is used during a viewing phase. One or more light metersmounted adjacent the camera 30 co-located with the viewer are operatedto monitor ambient and non-ambient luminance, chromaticity and possiblyother light parameter levels. Control signals are generated on the basisof the monitored levels of luminance and chromaticity, etc. The controlsignals are used to generate a real-time modification of the image dataused to generate the modifying image so that the modifying image and thefoundation images do not have luminance, chromaticity or other lightparameter levels which would cause a mismatch as between elements of themodifying image and elements of the foundation image when viewed as acombined image.

Video stream data transmitted to a viewer may be as a single dedicatedchannel which is demodulated and decoded by demodulation and decodingequipment forming part of the viewer's equipment. Alternatively, a setof video streams are selected from the totality of the available storedvideo streams based on select commands from, for example, a plurality ofviewers. The video stream data for the selected video streams is mixedinto a composite stream of video data which is transmitted via acommunication path to the viewers. Each viewer has a viewer moduleoperable to separate from the composite video data only the video dataappropriate to that viewer's vantage position and to the particularvideo selection made by that viewer.

It will be seen that videographics processing may be implemented in theproduction phase, the viewing phase, or in both the production andviewing phases. Videographics preprocessing by the addition and/or thesubstitution of a videographics component to/for the video data is done(or is done mainly) in the production phase. Conventional animationsoftware development tools may be used to identify object data and tolink corresponding objects in successive frames.

In each of the previously described embodiments, elements of thefoundation image and elements of the modifying image are described asbeing spatially combined at a display. Each of the embodiments can alsobe implemented with control circuitry which combines the foundation andmodifying images temporally. For example, frames of modifying image arealternated with frames of foundation image at the heads-up display at aframe rate sufficiently high that the combined image is seen as a singlevideo or videographics image of the modifying image elementssuperimposed on the foundation image elements.

1.-20. (canceled)
 21. A system, comprising: a heads-up viewing devicecomprising: one or more cameras configured to capture a foundationalimage of a scene; a range finder configured to determine one or moreranges of one or more objects in the scene; an image processorconfigured to: determine a first range of a first object in the sceneusing the range finder; determine a second range of the second object ina modifying image of the scene that corresponds to the foundationalimage; superimpose the second object from the modifying image onto thefoundational image based at least in part on the first and secondranges; and a display unit configured to display the foundational imagewith the superimposed second object.
 22. A system of claim 21, wherein aheads-up viewing device comprises a wearable device.
 23. The system ofclaim 21, wherein: the range finder and the image processor areimplemented using a central processing unit (CPU) of the heads-upviewing device; and to superimpose the second object onto thefoundational image, the image processor is configured to: select afragment of the second object to superimpose based at least in part onwhich of the first object or the second object is closer; and generate acombined image that combines the foundational image and the fragment ofthe second object.
 24. The system of claim 21, wherein the second rangeof the second object is determined based at least in part on rangemetadata associated with the modifying image, and the range metadata isreceived from a remote server.
 25. The system of claim 21, wherein thedisplay unit comprises a partially reflective screen, wherein thefoundational image in viewable through the partially reflective screen,and the second object is projected onto the partially reflective screen.26. A system of claim 21, wherein the heads-up viewing device isconfigured to generate video data of the scene as a user wearing theheads-up viewing device moves through the scene.
 27. A system of claim21, wherein the second object is artificially constructed by theheads-up viewing device.
 28. A system of claim 21, wherein the videodata is generated based at least in part on a position of a user of theheads-up viewing device, determined by a positional sensor of theheads-up viewing device.
 29. A system of claim 28, wherein the positionsensor is a Global Positioning System (GPS) sensor.
 30. A system ofclaim 21, wherein the video data is generated based at least in part ona head tilt of a user determined by a head position sensor of theheads-up viewing device.
 31. A system of claim 21, wherein the videodata is generated based at least in part on an eye gaze direction of auser determined by an eye gaze monitor of the heads-up viewing device.32. A system of claim 21, wherein the video data is generated based atleast in part on an eye focus of a user determined by a depth-of-focusmonitor of the heads-up viewing device.
 33. A system of claim 21,wherein the video data is generated based at least in part on a lightparameter determined by a light monitor of the heads-up viewing device.34. A system of claim 21, wherein the heads-up viewing device includes arate of change detector that monitors a rate of movement of a user ofthe heads-up viewing device, and the heads-up viewing device isconfigured to suspend display of the second object when the rate ofmovement exceeds a threshold.
 35. A system of claim 21, wherein theheads-up viewing device includes an audio player configured to playaudio data that corresponds to video data generated by the heads-upviewing device.
 36. A system of claim 35, wherein the heads-up viewingdevice includes a user interface configured to receive commands from auser of the heads-up viewing device, wherein the video data is generatedin accordance with the commands.
 37. A method, comprising: performing,by a heads-up viewing device: capturing, via one or more cameras, afoundational image of a scene; determining, via a range finder, a firstrange of a first object in the scene; executing an image processing,wherein the execution includes: determining a second range of the secondobject in a modifying image of the scene that corresponds to thefoundational image; superimposing the second object from the modifyingimage onto the foundational image based at least in part on the firstand second ranges; and displaying, via a display unit, the foundationalimage with the superimposed second object.
 38. The method of claim 37,wherein superimposing the second object onto the foundational imagecomprises: selecting a fragment of the second object to superimposebased at least in part on which of the first object or the second objectis closer; and generating a combined image that combines thefoundational image and the fragment of the second object.
 39. The methodof claim 37, wherein the display unit comprises a partially reflectivescreen through which the foundational image is viewable, and the methodfurther comprises the heads-up viewing device projecting the secondobject onto the partially reflective screen.
 40. A method of claim 37,further comprising the heads-up viewing device generating video data ofthe scene as a user wearing the heads-up viewing device moves throughthe scene.